Large-scale event evaluation using realtime processors

ABSTRACT

Large-scale event processing systems are often designed to perform data mining operations by storing a large set of events in a massive database, applying complex queries to the records of the events, and generating reports and notifications. However, because such queries are performed on very large data sets, the processing of the queries often introduces a significant delay between the occurrence of the events and the reporting or notification thereof. Instead, a large-scale event processing system may be devised as a large state machine organized according to an evaluation plan, comprising a graph of event processors that, in realtime, evaluate each event in an event stream to update an internal state of the event processor, and to perform responses when response conditions are met. The continuous monitoring and evaluation of the stream of events may therefore enable the event processing system to provide realtime responses and notifications to complex queries.

BACKGROUND

Within the field of computing, many scenarios involve a set of events tobe evaluated through data mining. As a first example, the events maycomprise the actions of a set of customers interacting with a commercialstore, website, product, or service, and the actions may be evaluated toidentify consumer trends. As a second example, the events may comprisethe actions of individuals comprising a demographic, a group, or anorganization, and the actions may be evaluated to identify patterns ofbehavior of behavior among the individuals. As a third example, theevents may comprise the actions of users who wish to receive servicesand information that may be of interest to the users. As a fourthexample, the events may comprise measurements of a system, such as amachine or an environment that are to be evaluated to monitor the stateof the system on behalf of an administrator. As a fifth example, theevents may comprise measurements performed in a technical or scientificstudy, and the evaluation may be performed to identify relevantinformation.

Many such processing systems are centered around a large database andthe evaluation of data stored therein. For example, many such processingsystems are designed as a server farm, comprising a large number ofdatabase servers interoperating as a distributed database, andconfigured to generate various queries to be applied to a very largedata set stored therein. This information may be stored, e.g., as alarge set of tables comprising interrelated records, where such tablesand records may be distributed across the database servers comprisingthe server farm. As an example of such large-scale processing, many suchdata processing systems utilize a MapReduce-based framework, wherein acentral coordinating system may evaluate a query by identifying variousquery components, distributing each query component to a databaseservers storing information relevant to the query component, andcompositing the query results generated by each database server togenerate a query response. Such databases are often designed to store alarge amount of data gathered over a period of time, and to apply largeand complex queries to large numbers of records (potentially comprisingbillions of records gathered over a long period of time), and theevaluation of such queries may eventually result in a result setcomprising portions of the records satisfying the criteria of the query.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key factors oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Event processing systems centrally designed around records stored in adatabase may present many advantages, e.g., a detailed evaluation ofnuances of events detected over a long period of time. However, suchdatabase systems may involve some disadvantages. First, the applicationof a query to a large data set may take a while to execute against thelarge data set. For example, reports based on various queries may beperiodically generated, but the frequency of such reports may introducea delay between the occurrence of an event and the results of thereport, thereby limiting the responsiveness to recent events byrecipients (e.g., individuals and processes) who receive the report. Asa second example, the administrative storage of a large volume ofhistoric information may be very costly, involving the administrationand costs of acquiring and maintaining large storage systems. As a thirdexample, the recipients of a report may have little interest in thefine-grain details of events stored in the database, but rather areinterested only in the high-level reports resulting from the evaluation.However, since the computation is performed after storing theinformation, the administrators may be unable to reduce the datafootprint without losing details captured in the records of the eventsthat influence the results of reports. Thus, many of the stored recordsmay be too specific to be of much direct value to the administrators(while also imposing a significant administrative cost), yet theadministrators may have to capture the information in order to receivethe results of analyses utilizing such records.

Presented herein are techniques for improving the evaluation of eventsby large-scale processors. In accordance with these techniques, theevaluation of such events is not principally performed by storingrecords of such events in a database, and then applying a query to thedatabase to generate a query result representing useful information.Rather, the processing system may be formulated as a large statemachine, comprising a graph of realtime event processors operating asindividual state machines and communicating with the other eventprocessors. An evaluation plan may be provided (e.g., a stepwise logicalprocess of evaluating a set of events), and for various evaluationstages of the evaluation plan, one or more event processors may beinstantiated, each event processor having a set of response conditionsthat, when satisfied, cause the event processor to perform a response.When an event is received, the event processor may perform someprocessing on the event, update its internal state, and, if its responseconditions are satisfied, perform the response (e.g., generating areport or notification, or communicating the event or a processingresult to another event processor). Thus, the results and effects of theevents and processing results are stored not in a database, but in theinternal states of the state machines. Moreover, the monitoring andprocessing of a stream of events results in notifications and responsesto queries based on the current internal states of the state machines,which have been generated from and reflect the history of the eventstream. As a result, notifications and reports based on a long historyof accumulated knowledge may be generated promptly after the detectionof the event or the receipt of the query.

To the accomplishment of the foregoing and related ends, the followingdescription and annexed drawings set forth certain illustrative aspectsand implementations. These are indicative of but a few of the variousways in which one or more aspects may be employed. Other aspects,advantages, and novel features of the disclosure will become apparentfrom the following detailed description when considered in conjunctionwith the annexed drawings.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an exemplary scenario featuring aprocessing of events received from an event source using an eventprocessing database.

FIG. 2 is an illustration of an exemplary scenario featuring aprocessing of events received from an event source using a realtimeevent processing set according to the techniques presented herein.

FIG. 3 is a flow chart illustrating an exemplary method of evaluatingevents received from an event source.

FIG. 4 is a component block diagram illustrating an exemplary system forevaluating events received from an event source.

FIG. 5 is an illustration of an exemplary computer-readable mediumcomprising processor-executable instructions configured to embody one ormore of the provisions set forth herein.

FIG. 6 illustrates an exemplary computing environment wherein one ormore of the provisions set forth herein may be implemented.

DETAILED DESCRIPTION

The claimed subject matter is now described with reference to thedrawings, wherein like reference numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of the claimed subject matter. It may beevident, however, that the claimed subject matter may be practicedwithout these specific details. In other instances, structures anddevices are shown in block diagram form in order to facilitatedescribing the claimed subject matter.

Within the field of computing, many scenarios involve the processing ofa large set of events received from an event source. The events maycomprise, e.g., the actions of individuals, measurements of physicalproperties, or reports generated by automated processes. As a firstexample, the events of a set of consumers may be detected to observeconsumer trends, such as the popularity of products and services, userexperiences, and consumer decisions. As a second example, the events ofa set of individuals, such as members of a community or an organization,may be detected to identify behavioral patterns, such as evaluatinginteractions and discussions occurring within a social network. As athird example, an end user may utilize a computer to track his or heractions in order to identify interests, present information relatedthereto, and take actions on behalf of the end user. As a fourthexample, measurements of a system, such as a natural environment, amachine, or a processing plant, may be recorded to evaluate, analyze,and monitor the state of the system. As a fifth example, the events mayarise within a field of technical or scientific research, such asmedical events of individuals in a community, and may be evaluated togenerate new information within the domain of the research. In these andother scenarios, a large set of data about the events may be generatedand reported to a processing system for processing via data miningtechniques. The evaluation may be directed, e.g., formulated by anadministrator to retrieve information in response to specific queries,and/or undirected, e.g., automatically identifying significant patternswithin the events that may be of interest to the administrator, such asclustering similar types of data or formulating knowledge or heuristicsabout the information domain of the events.

Many such systems are designed as a set of computers, such as a serverfarm, that together manages a large and complex database configured tostore a large number of records (possibly numbering into the billions).For example, the database may be distributed over a large number ofdatabase servers, each configured to store a portion of the database(e.g., a subset of the tables of the database, and/or a subset of therecords of one or more tables), and to apply various queries to therecords stored therein. The database may also be structured as a seriesof layers, e.g., a first set of servers operating as a data storagelayer configured to store the data, and a second set of serversoperating as a business logic layer configured to access the data andperform complex evaluation of the data. These servers may support manytypes of operations, e.g., complex and historic analyses of the data,monitoring for events of particular interest, providing reports thatsummarize the data in various ways, and data mining to identify patternsand knowledge represented within the database.

One such technique that is often utilized in such event processingsystems is the distribution of a query over a set of database servers.The MapReduce framework is often utilized for this task, and does so byproviding a master node having information about the other databaseservers comprising the event processing system. Upon receiving a query,the master node may seek to decompose the query into various subqueries,each pertaining to a subset of data that is stored on a particularsubset of database servers. For example, if a query involves a combinedevaluation of three tables, the master node may seek to find one or moredatabase servers that are currently storing the respective tables, andmay send a subquery to each database server querying the data storedthereby. Upon receiving the results of the query evaluation from eachdatabase server, the master node may compose the results into a resultset that is responsive to the query, and may provide it to theindividual or process that submitted the query.

FIG. 1 presents an illustration of an exemplary scenario 10 featuring aset of computers operating together as an event processing database 16.In this exemplary scenario 10, an event source 12 generates a series ofevents 14 that are reported to the event processing database 16 forevaluation. A query processor 18 of the event processing database 16 mayaccept the events 14, identify one or more database servers 20 storingthe event 14, and provide the event 14 to the identified databaseserver(s) 20 for storage in the database. Additionally, when a user 22submits a query 24, the query processor 18 may receive the query 18 andevaluate it to identify various subqueries 26 involving various portionsof the database (e.g., a join query specifying a selection from firsttable and a selection from a second table) and the database servers 20storing such portions. The query processor 18 may then send thesubqueries 26 to one or more database servers 20 for evaluation. Eachdatabase server 20 may receive a subquery 26, process it against thelocally stored portion of the database, and return a response 28, suchas a selection or projection of data. The query processor 18 may thenreceive the responses 28 and composite a set of query results 30 toreturn to the user 22. In this manner, the event processing database 16may scale up to handle the storage of a vast number of records (e.g.,billions of records of various events 14) and the application of complexqueries to achieve sophisticated analyses. Additionally, data miningtechniques may be applied to identify new information represented in theevents, such as patterns of events 14, classifications and clusters ofevents 14, and heuristics.

While event processing database 16 in the exemplary scenario 10 of FIG.1 presents some advantages, some disadvantages may also arise from thedesign of the processing system around a large database. As a firstexample, the storage of a large database involves significant cost inacquiring the storage and processing hardware, and in configuring andadministrating the event processing system 16. For example,administrators may expend considerable attention in designing,implementing, securing, and maintaining the data stores, the databasecontained therein, and the interrelationships of the query processor 18and the database servers 20. Moreover, the storage of a large number ofevents 14 with a high degree of granularity may not be cost-effective.For example, a complex query applied against the database may generatehigh-level results (e.g., macroanalyses and extracted knowledge) fromthe low-level records stored in the database, but otherwise, the storageof voluminous low-level data may impose a significant administrativecost without significant benefit. Yet, it may not be feasible to cullthe voluminous low-level data, since the execution of future queries 24may be skewed if not applied against older events 14 as well as newerevents 14.

As a second example, the processing of any query 24 against the largenumber of records stored in the database may involve considerablecomputational resources, since the queries may potentially be executedover a large number and/or complexity of records. As a result, theprocessing of each query 24 may involve a considerable expenditure ofcomputing resources and a considerable delay. Accordingly, complexqueries 24 are often executed only periodically, e.g., once per week.Additionally, the event processing database 16 may be unable to monitorevents 12 continuously for updates or changes. As a result, the eventprocessing database 16 may be unable to provide continuous, realtimereports and notifications in response to newly received events 14. Thedelay imposed between the receipt of events 12 and the notification orreporting of query results 30 to a user 22 may diminish the value ofthereof. For example, if a complex query is executed weekly, a user 22may not be notified of events 14 until up to a week after occurrence.This delay may be costly in circumstances where rapid responsiveness maybe of significant value.

As a third example, an event processing database 16 may not be able tocontinue evaluating a query 24 against events 14 in a continuous manner.In some scenarios, the event processing database 16 may, afterevaluating a query 24 and generating query results 30, apply the query24 to subsequently received events 14 and supplement the results withnewly generated records based on such events 14. However, thissupplemental approach does not holistically reevaluate the query 24against the entire database. For example, in a data mining techniquethat involves a clustering of events 14 into distinctive types, a firstevaluation of the query 24 may identify some clusters of events 14, andcontinued evaluation of newly received events 14 may classify events 14into previously identified clusters, and may also identify new clusterscomprising events 14 that are markedly different from the previouslyidentified clusters. However, the supplemental evaluation of new eventscannot reassess the previously identified clusters in view of the newevents 14 without reevaluating the query 24 holistically against theentire database.

This inefficiency may arise, in significant part, from the statelessnessof the event processing database 16 in the evaluation of each query 24anew. For example, the processing of a query 24 involves the current setof records and the operations specified in the query 24, but does notinclude knowledge or previous computations arising from previouslygenerated queries 24. For example, executing the same query 24repeatedly on the database involves gathering the same records andperforming the same evaluation thereupon. Instead, the event processingdatabase 16 reevaluating the referenced data set from scratch for eachquery 16. While caching techniques may improve the throughput of thequery 24, the results of one query 24 do not meaningfully inform theresults of further queries 24. In general, the event processing database16 stores information and naïvely evaluates queries 24, but does notcapture or utilize knowledge. Additionally, fine-grain data within thedatabase that has already been evaluated may have to be preserved inorder to achieve consistent results for future queries 24, because theevaluation of the query 14 can only utilize previously discoveredknowledge by reprocessing the previously evaluated events 14 along withthe subsequently received events 14. For example, in scenarios involve aclustering of events 14, the query 24 may differentiate events 14 intoclusters based on heuristics discovered during the query 24. However,when the query 24 terminates, the heuristics may be discarded—e.g., thedatabase may preserve the query results 30 (e.g., the identifiedclusters and the events 14 related thereto), but not the rationalewhereby the clusters were selected. Accordingly, these rationale may notbe available to the event processing database 16 while evaluatingsubsequently received events 14. Reevaluation may only be achievable byre-executing the entire query 24 against the entire database. Thisnaïvete in the evaluation of events 14 may present significantlimitations in the capabilities of the event processing system. Forexample, for complex queries 24, involving a large number of databaserecords and/or complex processing of such records, continuous andrealtime evaluation, or even frequent evaluation, may becost-ineffective or simply unachievable. These and other disadvantagesmay arise from the implicit inefficiencies in using a large database toevaluate the events 14.

Presented herein are alternative techniques for evaluating events 14received from an event source 12. These techniques involve modeling anevent processing system not as a database to be naïvely queried for eachquery 24, but rather as a state machine. In these techniques, anevaluation plan is provided involving one or more evaluation stagesinvolved in the evaluation of the events 14. The evaluation plan may begenerated by a user 22 (such as an administrator) with or without theassistance of a computer. Using this evaluation plan, an event processorset may be generated, comprising event processors configured to performan evaluation of events 14 for each evaluation stage of the evaluationplan. Each event processor may have an internal state, and uponcompleting an evaluation of an event 14, may update its internal statewith the results of the evaluation. The event processor may thereforestore knowledge extracted from the evaluation of events 14, and mayretain and utilize that knowledge while evaluating subsequent events 14.Therefore, by comparing its internal state upon evaluating each event 14and making adjustments, each event processor may retain the knowledgeextracted during each evaluation, and may re-use and reexamine thisknowledge while processing future events 14. Moreover, each eventprocessor may also perform one or more responses based on whether or notthe state of the of the event processor matches particular responseconditions. For example, an event processor may notify a user 22 oradministrator upon identifying significant changes in the state of theevent processor, which may represent knowledge or significantinformation. Moreover, event processors may be interconnected, e.g.,with event processors intercommunicating in response to the evaluationof events 14. Finally, the event processor set may be connected to theevent source to initiate the monitoring of the events 14 generatedthereby. Because the evaluation of each event 14 (including newlysubmitted queries 24) involves the examination of the internal states ofthe event processors instead of a naïve application of the query to theentire set of events 14 stored in the database, the event processors maybe able to evaluate and respond to events 14 in a rapid manner.Accordingly, such evaluation may be performed continuously on an eventstream. This may result in notifications, reports, and query results 30that may be delivered in realtime, including both the latest events 14in view of the knowledge extracted from the evaluation of all precedingevents 14.

As one such example, an event processor set may be configured toidentify patterns of individual preferences in movies that may berevealed through events 14 comprising the interactions of theindividuals with a movie library. Individuals may perform variousactions in such interactions, including searching for movies, browsingthe titles and descriptions of movies, responding to recommendations ofmovies, and selecting movies for viewing, as well as actions performedwhile viewing the movies, such as pausing, skipping ahead or back,viewing again, and terminating without completing a viewing the movie.One individual may therefore generate many such events 14 during aviewing session, and thousands of individuals may generate a voluminousstream of such events 14. In accordance with the techniques presentedherein, an evaluation plan may be devised (e.g., by an administrator ofthe event processing system, with or without the assistance of acomputer) comprising various evaluation stages that may be applied tothis task. Such evaluation stages may include, e.g., inferring theinterests of an individual from the searches and selections of movies;clustering and classifying individuals based on demographics, inferredinterests, and viewing history; and identifying heuristics that may beindicative of interests, such as actions that are highly correlated withhigh interest or low interest. This evaluation plan may then beimplemented by automatically generating one or more event processors,each tasked with performing some or all of the evaluation for aparticular evaluation stage. These event processors may then beinterconnected to generate an event processor set that covers theevaluation stages of the evaluation plan. Finally, the event processorset may be connected to an event source, which may begin delivering acontinuous stream of events 14 to the event processor set. Theevaluation of each event 14 is rapid, e.g., because each event processorwithin the event processor set may perform a comparatively simpleevaluation based on the event 14 and its current state. As a result, theevent processor set may scale with acceptable performance to handle alarge and continuous volume of events 14, thereby enabling a realtimegeneration of reports, notifications, and query results 30 in responseto new information and knowledge identified in the stream of events 14,such as anomalies and developing trends.

FIG. 2 presents an illustration of an exemplary scenario 40 featuring anevent processor set 44 configured to evaluate a stream of events 14received from an event source 12 in accordance with the techniquespresented herein. In this exemplary scenario 40, an evaluation plan 42may be provided that comprises a set of evaluation stages to be appliedto the events 14. An embodiment of these techniques may utilize theevaluation plan 42 to generate one or more event processors 46 forrespective events stages of the event plan, each maintaining a state 48that may be adjusted in view of information and knowledge extracted fromthe continuous evaluation of events 14. The event processors 46 may beinterconnected to form an event processor set 44 reflecting theevaluation embodied in the evaluation plan 42. The event processor set44 may then be connected to the event source 12, and may begincontinuous evaluation of the stream of events 14 generated by the eventsource 12. Upon evaluating the events 14, respective event processors 46may update the state 48 maintained therein, and may, if the state 48satisfies a response condition, generate a response. For example, anevent processor 46 may generate and deliver a query result in responseto a query 24 submitted by a user 22, or may generate and send the user22 a notification 50 (e.g., a notification 50 of a newly identifiedtrend among the events 14, or of significant changes to the state 48 ofthe event processor 46 that may represent significant new information orknowledge). The event processors 46 may also perform other responses,e.g., generating a report 52 or storing information in a database 54(e.g., recording significant events 14 in a database for that may thenbe subjected to traditional data mining techniques). These and otherresponses may be achieved in realtime, due to the continuous evaluationof events 14 from the event source 12 and the persistence of knowledgebetween evaluations, and may scale to handle a large and continuousvolume of events 14 due to the rapid evaluation achievable byimplementing the event processor set 44 as a state machine. These andother advantages may be achievable through the application of thetechniques presented herein.

FIG. 3 presents a first embodiment of these techniques, illustrated asan exemplary method 60 of evaluating events 14 received from an eventsource 12 according to an evaluation plan 42. The exemplary method 60may be implemented, e.g., as a set of software instructions stored in amemory component (e.g., a system memory circuit, a platter of a harddisk drive, a solid state storage device, or a magnetic or optical disc)of a device having a processor, that, when executed by the processor ofthe device, cause the processor to perform the techniques presentedherein. The exemplary method 60 begins at 62 and involves executing 64the instructions on the processor. More specifically, the instructionsare configured to, for respective evaluation stages of the evaluationplan 42, generate 66 an event processor 46 having a state 48 and atleast one response having a response condition. The event processor 46is also configured to, upon receiving an event 14, update 68 the stateof the event processor 46; and upon the state 48 of the event processor44 satisfying a response condition, perform 70 the response. Theinstructions are also configured to interconnect 72 the event processors46 to generate an event processor set 44 covering the evaluation plan42. The instructions are also configured to connect 74 the eventprocessor set 44 to the event source 12, thereby initiating theevaluation of the events 14. In this manner, the exemplary method 60achieves the evaluation of the events 14 according to the techniquespresented herein, and so ends at 76.

FIG. 4 presents a second embodiment of these techniques, illustrated asan exemplary system 86 configured to evaluate events 14 received from anevent source 12. The exemplary system 86 may be implemented, e.g., as asoftware architecture, comprising a set of components, each comprising aset of software instructions stored in a memory component (e.g., asystem memory circuit, a platter of a hard disk drive, a solid statestorage device, or a magnetic or optical disc) of a device 82 having aprocessor 84, such that, when executed (concurrently or consecutively)by the processor 84 of the device 82, cause the processor 84 to performone or more tasks of the techniques presented herein. The exemplarysystem 86 includes an evaluation plan 88 specifying at least oneevaluation stage of an evaluation of the events 14. The exemplary system86 also includes an event processor generating component 90, which isconfigured to, for respective evaluation stages of the evaluation plan88, generate an event processor 46 having a state 48 and at least oneresponse having a response condition. The event processor 46 is furtherconfigured to, upon receiving an event 14, update the state 48 of theevent processor 46; and upon the state 48 of the event processor 46satisfying a response condition, perform the response. The exemplarysystem 86 also includes an event processor set generating component 92,which is configured to interconnect the event processors 46 to generatean event processor set 44 covering the evaluation plan 88, and connectthe event processor set 44 to the event source 12. In this manner, theexemplary system 88 of FIG. 4 achieves the evaluation of events 14received from the event source 12 according to the evaluation plan 88.

Still another embodiment involves a computer-readable medium comprisingprocessor-executable instructions configured to apply the techniquespresented herein. Such computer-readable media may include, e.g.,computer-readable storage media involving a tangible device, such as amemory semiconductor (e.g., a semiconductor utilizing static randomaccess memory (SRAM), dynamic random access memory (DRAM), and/orsynchronous dynamic random access memory (SDRAM) technologies), aplatter of a hard disk drive, a flash memory device, or a magnetic oroptical disc (such as a CD-R, DVD-R, or floppy disc), encoding a set ofcomputer-readable instructions that, when executed by a processor of adevice, cause the device to implement the techniques presented herein.Such computer-readable media may also include (as a class oftechnologies that are distinct from computer-readable storage media)various types of communications media, such as a signal that may bepropagated through various physical phenomena (e.g., an electromagneticsignal, a sound wave signal, or an optical signal) and in various wiredscenarios (e.g., via an Ethernet or fiber optic cable) and/or wirelessscenarios (e.g., a wireless local area network (WLAN) such as WiFi, apersonal area network (PAN) such as Bluetooth, or a cellular or radionetwork), and which encodes a set of computer-readable instructionsthat, when executed by a processor of a device, cause the device toimplement the techniques presented herein.

An exemplary computer-readable medium that may be devised in these waysis illustrated in FIG. 5, wherein the implementation 100 comprises acomputer-readable medium 102 (e.g., a CD-R, DVD-R, or a platter of ahard disk drive), on which is encoded computer-readable data 104. Thiscomputer-readable data 104 in turn comprises a set of computerinstructions 106 configured to operate according to the principles setforth herein. In one such embodiment, the processor-executableinstructions 106 may be configured to perform a method of evaluatingevents received from an event source, such as the exemplary method 60 ofFIG. 3. In another such embodiment, the processor-executableinstructions 106 may be configured to implement a system for evaluatingevents received from an event source, such as the exemplary system 86 ofFIG. 4. Some embodiments of this computer-readable medium may comprise anon-transitory computer-readable storage medium (e.g., a hard diskdrive, an optical disc, or a flash memory device) that is configured tostore processor-executable instructions configured in this manner. Manysuch computer-readable media may be devised by those of ordinary skillin the art that are configured to operate in accordance with thetechniques presented herein.

The techniques discussed herein may be devised with variations in manyaspects, and some variations may present additional advantages and/orreduce disadvantages with respect to other variations of these and othertechniques. Moreover, some variations may be implemented in combination,and some combinations may feature additional advantages and/or reduceddisadvantages through synergistic cooperation. The variations may beincorporated in various embodiments (e.g., the exemplary method 60 ofFIG. 3 and the exemplary system 86 of FIG. 4) to confer individualand/or synergistic advantages upon such embodiments.

A first aspect that may vary among embodiments of these techniquesrelates to the scenarios wherein such techniques may be utilized. As afirst example, these techniques may be utilized to evaluate many typesof events 14, such as events generated by individuals, organizations, orautomated processes. As a second example, these techniques may becombined with traditional data mining techniques. For example, inaddition to performing responses, the event processors 46 may also storedata in various databases, to which traditional data processingtechniques may be utilized to extract additional information andknowledge. In turn, this information and knowledge may be utilized toupdate the state 48 of one or more event processors 46, and/or to updatethe evaluation plan 42, e.g., by adding or altering one or moreevaluation stages thereof. As a third example, many architectures may beutilized to implement these techniques. For example, the respectiveevent processors 46 may be implemented as separate machines, and/or astwo or more software processes executing on one or more such machines.Those of ordinary skill in the art may devise many scenarios wherein thetechniques presented herein may be utilized.

A second aspect that may vary among embodiments of these techniquesrelates to the generation of the evaluation plan 42. The evaluation plan42 may be generated completely by one or more users, such as anadministrator of the event processor set 44, and provided to anembodiment of these techniques (such as the exemplary system 86 of FIG.4) to generate the event processor set 44 and the event processors 46therein. However, in other scenarios, the evaluation plan 42 may begenerated with the assistance of an embodiment of these techniques. In afirst such variation, an embodiment may suggest aspects of theevaluation plan 42 that the user may wish to incorporate. For example,data mining techniques may be applied to a database of events 14 (eithergenerated by another technique or generated by the event processor set44) to extract knowledge and heuristics that may be suggested to theadministrator to include in an evaluation plan 42. As one such example,in a scenario where the event processor set 44 is provided to evaluateindividual interests in movies, data mining techniques applied to adatabase of events 14 representing the actions of individualsinteracting with a movie library may indicate that viewing a movie morethan twice is a strong indicator of individual interest, and that thistype of event 14 may be advantageously included as an evaluation stagewithin the evaluation plan 42.

As a second variation of this second aspect, an embodiment of thesetechniques may endeavor to automate part of the generation of theevaluation plan 42. In particular, a user may identify one or moreevaluation goals of the evaluation plan 42—e.g., the types ofinformation, knowledge, and heuristics that the user endeavors togenerate with the event processors set 44—and an embodiment of thesetechniques may automatically generate some parts of the evaluation plan42 in furtherance of these evaluation goals. As a first such example,the user may specify a particular topic (e.g., the popularity of a moviestar within a social network), and an embodiment of these techniques mayautomatically generate some parts of an evaluation plan 42 thatidentifies events 14 associated with the topic (e.g., identifyingmessages posted within the social network regarding the movie star,links to photos and articles about the website of the movie star, andreviews of the movies of the movie star, as well as an evaluation of thecorrelative weight of each type of event 14 to this topic). As a secondsuch example, the user may specify a topic (such as a product), and anembodiment of these techniques may automatically generate parts of anevaluation plan 42 that identify other topics that are related to thetopic (e.g., monitoring a commercial market to identify competingproducts and the popularity of such competing products among users). Asa third such example, the user may specify an evaluation result to bederived from the events 14 (e.g., a type of report or data set, such asa general summary of topics discussed in a social network), and anembodiment of these techniques may automatically generate parts of anevaluation plan 42 whereby event processors 46 are utilized to generatethe evaluation result. In these and other variations, an embodiment ofthese techniques may endeavor to assist an individual in generating theevaluation plan 44 in accordance with the techniques presented herein.

A third aspect that may vary among embodiments of these techniquesrelates to the operation of the event processors 46 of the eventprocessor set 44 to evaluate the events 14. As a first variation, theevent processors 46 may be configured, e.g., to evaluate respectiveevents 14 according to a particular task to be achieved by theevaluation stage wherein the event processor 46 is positioned (e.g., bytransforming incoming data into more useful outgoing data, e.g., bytranslating a foreign language message in a social network into a locallanguage for easier processing by other event processors 46).Alternatively, the event processors 46 may be configured to shape theevaluation of events 14 through the event processor set 44. As one suchexample, the event processors 46 may be configured in order to performtaxonomy reduction, e.g., by filtering or summarizing events 14 suchthat the number of events 14 in an incoming event set that the eventprocessor 46 receives and evaluates is larger the number of events 14 inthe outgoing events set that the event processor 46 transmits (e.g., toanother event processor 46, into a report or notification, or into adatabase), thereby reducing the amount of data exchanged and improvingthe significance of such data.

As a second variation of this third aspect, the types of responsesperformed by the event processors 46 may vary. As a first example, theevent processors 46 may provide realtime notifications 50 of theinformation gleaned from the events 14, reports 52, and/or realtimequery results 30 of one or more queries 24 submitted by a user 22.Alternatively or additionally, the event processors 46 may generate suchnotifications 50, reports 52, and/or query results 30 in a non-realtimemanner, e.g., by periodically evaluating a set of events 14 in adeferred time frame. As a second example, the responses of one or moreevent processors 46 may involve storing a record of at least one event14 (or a processing result of having evaluated such events 14) in atraditional data store, such as a database, for further evaluation byusers 22 and/or other processes.

A third variation of this third aspect relates to the manner of savingthe state of the event processor set 42. For example, one interestingcharacteristic of the event processor set 42 is that the informationgleaned from the evaluation of events 14 is stored in the states 46 ofthe respective event processors 46; however, this characteristic alsoraises a potential vulnerability to the loss of such information, e.g.,upon a machine or software process crashing. Therefore, it may bedesirable to capture a snapshot of the states of 48 the event processors46 of the event processor set 44, e.g., in order to evaluate theinformation contained in the event processor set 44 and/or to enable theevent processor set 44 to be restored to this state at a later time.Moreover, it may be desirable to refrain from stopping the eventprocessor set 44 from evaluating events 14 while capturing the snapshot,particularly where the event source 12 provides a continuous and heavystream of events 14. Therefore, an embodiment of these techniques maystore an event processor set snapshot, comprising the state 48 of eachevent processor 46 of the event processor set 44, while the eventprocessor set 44 continues to operate.

A fourth variation of this third aspect relates to the manner ofallowing users 22 to interact with the event processor set 44. Forexample, where the event processor set 44 is utilized to evaluate andreport on the actions of users 22, it may be undesirable to includepersonally identifying information (PII) of any particular user 22 inany such report, such that the privacy and trust of the user 22 andothers may be jeopardized. To this end, it may be advantageous to permitthe user 22 to examine the information that has been captured about theuser 22 and how such information is to be used various in variousreports 52. Therefore, when an embodiment of these techniques receives apresentation request to present the state 46 of various processors 44that involve the user 22, the embodiment may be configured to identifysuch event processors 46 and present the state 48 thereof to the user22. Additionally, if the user 22 submits an adjustment request to adjustthe state 48 of one or more event processors 46 (e.g., requesting tohave some sensitive or personally identifying information removed), theembodiment may accordingly alter the state 48 of the event processor(s)46. In this manner, the embodiment may extend to the users 22 monitoredby the event processor set 44 a measure of review and/or control overthe personal information that is collected and reported. Those ofordinary skill in the art may devise many ways of adjusting theexecution of the event processor set 44 in accordance with thetechniques presented herein.

A fourth aspect that may vary among embodiments of these techniquesrelates to updates to the evaluation plan 46 that may affect theconfiguration of the event processor set 44, e.g., the addition,removal, repositioning, and/or reconfiguring of one or more eventprocessors 46. For example, a user 22, such as an administrator, mayrequest to alter the evaluation plan 46 to adjust the details of theevaluation of events 14 and the responses generated by such eventprocessors 46. Alternatively, the evaluation plan 46 may be informed bythe results of the evaluation; e.g., new heuristics identified aspotentially valuable in the evaluation of the events 14 may beincorporated as new or improved event processors 46. Therefore, anembodiment of these techniques may be configured to, upon receiving anupdated evaluation plan, reconfigure the event processor set 44 to matchthe updated evaluation plan. However, it may be desirable to reduce theimpact of the reconfiguration upon the continued operation of the eventprocessor set 44, e.g., to permit the event processor set 44 to continueevaluating events 14 during the reconfiguration. Therefore, as a firstvariation of this fourth aspect, an embodiment of these techniques maybe configured to identify one or more updated event processors that areaffects by the updated evaluation plan 46, and, while the eventprocessor set 44 remains connected to the event source 12 and continuesevaluating the events 14, update only the updated event processors.

A second variation of this fourth aspect relates to the manner ofupdating the event processor set 44 with an updated evaluation plan 42to evaluate new information. A particular problem may arise in thisscenario, wherein the event processors 46 may not yet be informed of howto perform the evaluation (having not yet evaluated events 14 in thismanner), and therefor being unable to generate accurate results untilthe state 48 of such event processors 46 is adapted to the newevaluation expressed in the updated evaluation plan. For example, theevent processor set 44 may be newly tasked with identifying trends inthe popularity of one or more topics in a social network, but withouthaving been informed yet of the current popularity of such topics, theevent processor set 44 may generate anomalous results until suchfamiliarity is established. This “cold start” problem may be alleviated,e.g., by connecting the event processor set 44 to a historic event set,comprising a set of historic events that have previously been captured.For example, a data store of events 14 may be captured in a periodpreceding the updated evaluation plan, and may be run through the eventprocessor set 44 (e.g., while suppressing output of the results untilacceptable results are achieved) in order to achieve states 48 of theevent processors 46 that generate acceptable results.

However, the training of the event processor set 44 with a historicevent set may be costly; e.g., a large and varied set of historic eventsmay be available (e.g., if such historic events have been captured in adata warehouse, such as in a traditional data mining scenario), andtraining the event processor set 44 with a large set of historic eventsmay cause a delay in the availability of the event processing.Therefore, it may be desirable to reduce the size of the historic eventset in order to achieve a training of the event processors 46 with highefficiency and/or reduced cost. As a first such example, where a largenumber of historic events may be available from a historic event source(e.g., a large database or capture of historic events), the historicevents of the historic event set may be selected only in relation to theupdated aspects of the evaluation plan 42, e.g., only including historicevents 42 that are likely to be evaluated by one or more updated eventprocessors 46. As a second such example, an embodiment of thesetechniques may limit the historic event set to a selection of historicevents that match one or more historic event parameters (e.g., historicevents occurring within a designated period, such as the week precedingthe updating of the event processor set 44, or historic events involvingone or more users 22 and/or topics). Such historic event parameters maybe identified by a user 22, such as an administrator, or may beautomatically selected by an embodiment of these techniques, e.g., byidentifying historic event parameters that are likely to identifyhistoric events of high training value to the event processor set 46.

As an additional variation, an embodiment of these techniques may beable to estimate a computational cost arising from connecting the eventprocessor set 46 to a particular historic event set (e.g., an amount oftime that will be involved in applying the event processor set toevaluate the historic event set, or an amount of storage space orbandwidth utilized in such training). The computational cost may bepresented to a user 22, such as an administrator, to estimate the impactof training with a selected historic event set upon the functionality ofthe event processor set 44 (e.g., an amount of downtime of the eventprocessor set 44 that may be caused by the training). The embodiment maytherefore wait for the user 22 to verify the use of the historic eventset in view of the estimated computational cost. Additionally, the user22 may be permitted to adjust various historic event set parameters(e.g., the period and/or scope of historic events retrieved from thehistoric event source), to view the estimated computational costs oftraining with different historic event sets, and to select a historicevent set that offers an acceptable tradeoff of training value andcomputational costs. In this manner, an embodiment of these techniquesmay facilitate an administrator or other user 22 in achieving a trainingof an updated event processor set 44 with a historic event set, therebyreducing the impact of the “cold start” problem on the utility of theresults of the evaluation, while also managing the computational costsof such training. Those of ordinary skill in the art may devise manyways of updating the event processor set 44 in accordance with thetechniques presented herein.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

As used in this application, the terms “component,” “module,” “system”,“interface”, and the like are generally intended to refer to acomputer-related entity, either hardware, a combination of hardware andsoftware, software, or software in execution. For example, a componentmay be, but is not limited to being, a process running on a processor, aprocessor, an object, an executable, a thread of execution, a program,and/or a computer. By way of illustration, both an application runningon a controller and the controller can be a component. One or morecomponents may reside within a process and/or thread of execution and acomponent may be localized on one computer and/or distributed betweentwo or more computers.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedsubject matter. The term “article of manufacture” as used herein isintended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. Of course, those skilled inthe art will recognize many modifications may be made to thisconfiguration without departing from the scope or spirit of the claimedsubject matter.

FIG. 6 and the following discussion provide a brief, general descriptionof a suitable computing environment to implement embodiments of one ormore of the provisions set forth herein. The operating environment ofFIG. 6 is only one example of a suitable operating environment and isnot intended to suggest any limitation as to the scope of use orfunctionality of the operating environment. Example computing devicesinclude, but are not limited to, personal computers, server computers,hand-held or laptop devices, mobile devices (such as mobile phones,Personal Digital Assistants (PDAs), media players, and the like),multiprocessor systems, consumer electronics, mini computers, mainframecomputers, distributed computing environments that include any of theabove systems or devices, and the like.

Although not required, embodiments are described in the general contextof “computer readable instructions” being executed by one or morecomputing devices. Computer readable instructions may be distributed viacomputer readable media (discussed below). Computer readableinstructions may be implemented as program modules, such as functions,objects, Application Programming Interfaces (APIs), data structures, andthe like, that perform particular tasks or implement particular abstractdata types. Typically, the functionality of the computer readableinstructions may be combined or distributed as desired in variousenvironments.

FIG. 6 illustrates an example of a system 110 comprising a computingdevice 112 configured to implement one or more embodiments providedherein. In one configuration, computing device 112 includes at least oneprocessing unit 116 and memory 118. Depending on the exact configurationand type of computing device, memory 118 may be volatile (such as RAM,for example), non-volatile (such as ROM, flash memory, etc., forexample) or some combination of the two. This configuration isillustrated in FIG. 6 by dashed line 114.

In other embodiments, device 112 may include additional features and/orfunctionality. For example, device 112 may also include additionalstorage (e.g., removable and/or non-removable) including, but notlimited to, magnetic storage, optical storage, and the like. Suchadditional storage is illustrated in FIG. 6 by storage 120. In oneembodiment, computer readable instructions to implement one or moreembodiments provided herein may be in storage 120. Storage 120 may alsostore other computer readable instructions to implement an operatingsystem, an application program, and the like. Computer readableinstructions may be loaded in memory 118 for execution by processingunit 116, for example.

The term “computer readable media” as used herein includes computerstorage media. Computer storage media includes volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer readableinstructions or other data. Memory 118 and storage 120 are examples ofcomputer storage media. Computer storage media includes, but is notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, Digital Versatile Disks (DVDs) or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium which can be used to storethe desired information and which can be accessed by device 112. Anysuch computer storage media may be part of device 112.

Device 112 may also include communication connection(s) 126 that allowsdevice 112 to communicate with other devices. Communicationconnection(s) 126 may include, but is not limited to, a modem, a NetworkInterface Card (NIC), an integrated network interface, a radio frequencytransmitter/receiver, an infrared port, a USB connection, or otherinterfaces for connecting computing device 112 to other computingdevices. Communication connection(s) 126 may include a wired connectionor a wireless connection. Communication connection(s) 126 may transmitand/or receive communication media.

The term “computer readable media” may include communication media.Communication media typically embodies computer readable instructions orother data in a “modulated data signal” such as a carrier wave or othertransport mechanism and includes any information delivery media. Theterm “modulated data signal” may include a signal that has one or moreof its characteristics set or changed in such a manner as to encodeinformation in the signal.

Device 112 may include input device(s) 124 such as keyboard, mouse, pen,voice input device, touch input device, infrared cameras, video inputdevices, and/or any other input device. Output device(s) 122 such as oneor more displays, speakers, printers, and/or any other output device mayalso be included in device 112. Input device(s) 124 and output device(s)122 may be connected to device 112 via a wired connection, wirelessconnection, or any combination thereof. In one embodiment, an inputdevice or an output device from another computing device may be used asinput device(s) 124 or output device(s) 122 for computing device 112.

Components of computing device 112 may be connected by variousinterconnects, such as a bus. Such interconnects may include aPeripheral Component Interconnect (PCI), such as PCI Express, aUniversal Serial Bus (USB), firewire (IEEE 1394), an optical busstructure, and the like. In another embodiment, components of computingdevice 112 may be interconnected by a network. For example, memory 118may be comprised of multiple physical memory units located in differentphysical locations interconnected by a network.

Those skilled in the art will realize that storage devices utilized tostore computer readable instructions may be distributed across anetwork. For example, a computing device 130 accessible via network 128may store computer readable instructions to implement one or moreembodiments provided herein. Computing device 112 may access computingdevice 130 and download a part or all of the computer readableinstructions for execution. Alternatively, computing device 112 maydownload pieces of the computer readable instructions, as needed, orsome instructions may be executed at computing device 112 and some atcomputing device 130.

Various operations of embodiments are provided herein. In oneembodiment, one or more of the operations described may constitutecomputer readable instructions stored on one or more computer readablemedia, which if executed by a computing device, will cause the computingdevice to perform the operations described. The order in which some orall of the operations are described should not be construed as to implythat these operations are necessarily order dependent. Alternativeordering will be appreciated by one skilled in the art having thebenefit of this description. Further, it will be understood that not alloperations are necessarily present in each embodiment provided herein.

Moreover, the word “exemplary” is used herein to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “exemplary” is not necessarily to be construed as advantageousover other aspects or designs. Rather, use of the word exemplary isintended to present concepts in a concrete fashion. As used in thisapplication, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or”. That is, unless specified otherwise, or clearfrom context, “X employs A or B” is intended to mean any of the naturalinclusive permutations. That is, if X employs A; X employs B; or Xemploys both A and B, then “X employs A or B” is satisfied under any ofthe foregoing instances. In addition, the articles “a” and “an” as usedin this application and the appended claims may generally be construedto mean “one or more” unless specified otherwise or clear from contextto be directed to a singular form.

Also, although the disclosure has been shown and described with respectto one or more implementations, equivalent alterations and modificationswill occur to others skilled in the art based upon a reading andunderstanding of this specification and the annexed drawings. Thedisclosure includes all such modifications and alterations and islimited only by the scope of the following claims. In particular regardto the various functions performed by the above described components(e.g., elements, resources, etc.), the terms used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., that is functionally equivalent), even though notstructurally equivalent to the disclosed structure which performs thefunction in the herein illustrated exemplary implementations of thedisclosure. In addition, while a particular feature of the disclosuremay have been disclosed with respect to only one of severalimplementations, such feature may be combined with one or more otherfeatures of the other implementations as may be desired and advantageousfor any given or particular application. Furthermore, to the extent thatthe terms “includes”, “having”, “has”, “with”, or variants thereof areused in either the detailed description or the claims, such terms areintended to be inclusive in a manner similar to the term “comprising.”

What is claimed is:
 1. A method of evaluating events received from anevent source according to an evaluation plan on a computer having aprocessor, the method comprising: executing on the computer instructionsconfigured to: for respective evaluation stages of the evaluation plan,generate an event processor having a state and at least one responsehaving a response condition and configured to: upon receiving an event,update the state of the event processor; and upon the state of the eventprocessor satisfying a response condition, perform the response;interconnect the event processors to generate an event processor setcovering the evaluation plan; connect the event processor set to theevent source; and upon receiving an adjustment request to adjust a stateof a selected event processor, adjust the state of the selected eventprocessor according to the adjustment request.
 2. The method of claim 1,respective event processors configured to, for respective events,generate an outgoing event set that is smaller than the incoming eventset.
 3. The method of claim 1, the response comprising: generating anotification of an event.
 4. The method of claim 1: at least one eventcomprising a query received from a user; and the response comprising:providing a report to the user in response to the query.
 5. The methodof claim 1: the computer having access to a data store; and the responsecomprising: storing at least one event in the data store.
 6. The methodof claim 1, the instructions configured to generate an evaluation planusing at least one evaluation goal specified by a user.
 7. The method ofclaim 6: the user specifying an evaluation result derived from theevents, and the evaluation goal comprising: evaluating the events togenerate the evaluation result.
 8. The method of claim 6: the userspecifying a topic, and the evaluation goal comprising: identifyingevents associated with the topic.
 9. The method of claim 6: the userspecifying a topic, and the evaluation goal comprising: identifyingother topics associated with the topic.
 10. The method of claim 1: theevent processor set configured to store at least one event in a datastore; and the instructions configured to: mine the data store toidentify at least one evaluation goal, and suggest the evaluation goalto the user.
 11. The method of claim 1, the instructions furtherconfigured to: upon receiving an event through the event source: storethe event in a historic event set, and evaluate the events with theevent processor set; and upon receiving an updated evaluation plan:reconfigure the event processor set according to the updated evaluationplan; and connect the event processor set to at least a portion of thehistoric event set.
 12. The method of claim 11, reconfiguring the eventprocessor set comprising: identifying at least one updated eventprocessor that is updated by the updated evaluation plan; and whileconnecting the event processor set to the event source, updating onlythe updated event processors.
 13. The method of claim 11: the updatedevaluation plan associated with at least one updated event processor;and connecting the event processor set to at least a portion of thehistoric event set comprising: identifying a selected portion of thehistoric event set comprising historic events associated with at leastone updated event processor; and connecting the event processor set tothe selected portion of the historic event set.
 14. The method of claim13, the instructions configured to generate the historic event set byselecting, from a historic event source, historic events matching atleast one historic event parameter.
 15. The method of claim 14, theinstructions configured to: estimate a computational cost arising fromconnecting the event processor set to the historic event set; and beforeconnecting the historic event set to the event processor set, verify thecomputational cost with a user.
 16. The method of claim 1, theinstructions configured to: upon receiving from a user a presentationrequest to present a state of at least one event processor in responseto an event involving the user, present the state of the eventprocessor; and upon receiving from a user an adjustment request toadjust the state of at least one event processor, adjust the state ofthe at least one event processor.
 17. The method of claim 1, theinstructions configured to store an event processor set snapshotcomprising the states of the event processors of the event processorset.
 18. A computer-readable storage device comprising instructionsthat, when executed on a processor of a computer, evaluate eventsreceived from an event source according to an evaluation plan by: forrespective evaluation stages of the evaluation plan, generating an eventprocessor having a state and at least one response having a responsecondition and configured to: upon receiving an event, updating the stateof the event processor; and upon the state of the event processorsatisfying a response condition, performing the response;interconnecting the event processors to generate an event processor setcovering the evaluation plan; connecting the event processor set to theevent source; and upon receiving an adjustment request to adjust a stateof a selected event processor, adjusting the state of the selected eventprocessor according to the adjustment request.
 19. The computer-readablestorage device of claim 18, the instructions configured to generate thehistoric event set by selecting, from a historic event source, historicevents matching at least one historic event parameter.
 20. A system forevaluating events received from an event source according to anevaluation plan on a device having a memory and a processor, the systemcomprising: an event processor generator comprising instructions storedin the memory that, when executed on the processor, for respectiveevaluation stages of the evaluation plan: generate an event processorhaving a state and at least one response having a response condition andconfigured to: upon receiving an event, updating the state of the eventprocessor; and upon the state of the event processor satisfying aresponse condition, performing the response; interconnecting the eventprocessors to generate an event processor set covering the evaluationplan; and connecting the event processor set to the event source; and anevent processor updater comprising instructions stored in the memorythat, when executed on the processor upon receiving an adjustmentrequest to adjust a state of a selected event processor, adjust thestate of the selected event processor according to the adjustmentrequest.